Paper: FINE-GRAINED AUDIO-VISUAL JOINT REPRESENTATIONS FOR MULTIMODAL LARGE LANGUAGE MODELS (ICLR-2024)

Reviewer comment (Presentation):
Very personal opinion for Fig 1: I think the trainable modules should have different colors to be even easier to understand the figure. All causal Q-former should have the same color, as they are applied on multiple windows with the same parameters. But then, the LLM should have a different color and each encoder should also have a different color. In this way you are making sure no one thinks you are using the same parameters across all these. Also a lot of papers use a special symbol (the fire symbol) for the trainable layers, as in video-llama figure and other papers which I think would help if you also used it.

Author response:
Thanks for pointing this out. We have added the fire symbol and made different blocks have different colours in Fig. 1.

Reviewer comment (Novelty):
The novelty is somewhat limited as I will argue below: Regarding the benchmarks, what is the novelty? As I understand, the paper does not provide a newly created dataset, but only puts together previously established datasets and calls it a new benchmark. Correct me if I am wrong.

Author response:
For weakness 2, we would like to first clearly list FAVOR’s contribution as follows:
  - FAVOR is the first single model that integrates audio, speech and video inputs, which is the most obvious difference between FAVOR and other multimodal LLMs. The incorporation of speech causes significant changes both to the approach itself and the extent of applications.

  - Unlike audio events, speech contains more temporal fine-grained information that is closely correlated with but very difficult to directly infer from video. This nature motivates the design of the FAVOR structure, including fine-grained (high temporal resolution) modelling, audio-visual synchronisation, causal attention module, and diversity loss.

  - Instead of having audio and video streams coming from the same source, FAVOR in addition supports audio from a different source which enables the training of the two modalities more balanced and also the performing of more versatile audio-visual co-reasoning tasks, including ISQA and AVM.

  - To the best of our knowledge, FAVOR is the first model that is able to perform open-ended audio-visual-speech-text questions, such as those given in Fig. 9 to 12 in the appendix.

- For each point of weakness 2:
> Regarding the benchmarks, what is the novelty? As I understand, the paper does not provide a newly created dataset, but only puts together previously established datasets and calls it a new benchmark. Correct me if I am wrong. 
  - We would like to clarify that the AVEB benchmark also contains newly designed tasks to reflect the audio-visual co-reasoning ability, including ISQA and AVM. These require the system to understand speech.

Reviewer comment (Novelty):
The novelty is somewhat limited as I will argue below: As I could see, the only difference between the proposed method and the video-LLAMA is using causal attention in the q-former, and the synchronization.

Author response:
> As I could see, the only difference between the proposed method and the video-LLAMA is using causal attention in the q-former, and the synchronization.
  - While FAVOR has some similarities to Video-LLaMA, we’d like to clarify the following superior aspects of FAVOR:

    - FAVOR is able to understand speech input which is unable to be handled by Video-LLaMA. 

    - Compared to Video-LLaMA, incorporating speech input motivated us to design this significantly different model structure of FAVOR, including the synchronisation, sliding window operation and causal Q-Former.

    - In contrast to sampling a fixed number of frames regardless of the video length in Video-LLaMA, FAVOR performs fine-grained modelling and the sliding window allows more output tokens for longer videos, which leads to significantly better performance.

  - __As a result, Video-LLaMA could not perform any of the examples shown in Fig. 6 to 12.__

Reviewer comment (Experiments):
Why some datasets were chosen only for training and some only for testing? In the paper, there are some explanations for some of them, but there are still datasets such as llava-150k, text caps, ego4d or videochat, that were chosen for either testing or training without mentioning why.

Author response:
> Why some datasets were chosen only for training and some only for testing? In the paper, there are some explanations for some of them, but there are still datasets such as llava-150k, text caps, ego4d or videochat, that were chosen for either testing or training without mentioning why.
  - Tasks including IC, OCR, VQA and AVSSD are set as zero-shot tasks for a fair comparison with InstructBLIP and Video-LLaMA. ISQA and AVM are set to be zero-shot tasks due to the limited data samples.

Reviewer comment (Experiments):
From section 4.2: “fair comparisons, InstructBLIP is further fine-tuned on the same image and video training data as FAVOR.” Was the same done for Whisper? If not, why? If yes, mention it.

Author response:
> From section 4.2 :“fair comparisons, InstructBLIP is further fine-tuned on the same image and video training data as FAVOR.” Was the same done for Whisper? If not, why? If yes, mention it.
  - We did not perform fine-tuning on Whisper as the ASR training data used here is small, and fine-tuning usually leads to degraded performance. The main aim of fine-tuning InstructBLIP is to eliminate the training data variable and fairly compare the structure advantage of FAVOR.

Reviewer comment (Experiments):
Does Video-LLAMA use the same exact visual/audio encoders? If not, I am afraid the results will not reflect the true capabilities of this model.

Author response:
> Does Video-LLAMA use the same exact visual/audio encoders? If not, I am afraid the results will not reflect the true capabilities of this model.Moreover, Video-LLAMA does not finetune the LLM, while this paper uses LoRA to finetune it. While one can argue that LoRA does not actually change the original weights, there is still some training happening there. Thus, I would also ask the authors to do an experiment where they keep the LLM frozen, as in video-llama and just train the other components. In this way it will be clear if the benefits come from the causal attention/synchronization model or if they come just from the fact that there are many more trainable parameters in the proposed model.
  - Regarding a fair comparison with Video-LLaMA, because incorporating speech is the key innovation, choosing a speech-based encoder is one of the important aspects of the model design. Therefore, it is reasonable to compare it to the Video-LLaMA in its original form although having a different audio encoder. However, we could still provide a point of comparison in response to your suggestions of modifying Video-LLaMA, by replacing the encoder, changing the video frame rate and using LoRA for training. This is the last row of Tables 8 and 9. The comparison to the last row clearly shows __over 10% improvements__ across all audio-visual tasks using synchronisation and causal Q-Former.

Reviewer comment (Experiments):
Moreover, Video-LLAMA does not finetune the LLM, while this paper uses LoRA to finetune it. While one can argue that LoRA does not actually change the original weights, there is still some training happening there. Thus, I would also ask the authors to do an experiment where they keep the LLM frozen, as in video-llama and just train the other components. In this way it will be clear if the benefits come from the causal attention/synchronization model or if they come just from the fact that there are many more trainable parameters in the proposed model.

Author response:
> In Table 2 and 3, some results are missing for the Video-LLAMA and are replaced with “-”. Why? As far as I understand, Video-LLAMA could be applied on all the tasks that the proposed method can be applied to by just removing the missing modality. One can just remove the audio or the visual branch in the Video-LLAMA and turn it into a uni-modal model. One would not even need to re-train it as the LLM is not trained at all, so it does not “care” if there are 2 modalities as input or only one. Please, correct me if I am wrong.
  - Video-LLaMA can not perform ASR and AVSR (WER over 100%), and hence also ISQA. Video-LLaMA is not trained to support single audio input, and hence single audio input will confuse the model.

Reviewer comment (Experiments):
In Table 2 and 3, some results are missing for the Video-LLAMA and are replaced with “-”. Why? As far as I understand, Video-LLAMA could be applied on all the tasks that the proposed method can be applied to by just removing the missing modality. One can just remove the audio or the visual branch in the Video-LLAMA and turn it into a uni-modal model. One would not even need to re-train it as the LLM is not trained at all, so it does not “care” if there are 2 modalities as input or only one. Please, correct me if I am wrong.

Author response:
> My main issue is the lack of novelty and the issue that the results may be good due to “unfair decisions”. Mainly, the authors need to clarify if both the proposed model and the video-llama use the same encoders or not. If they don’t use the same encoders, the experiments should be rerun to have a consistent setting with the same encoders.
  - As explained for weaknesses 3 to 6, we believe our experiments are fair and the required comparison point (including different encoders and LoRA) can be found in Tables 8 and 9.

Reviewer comment (Novelty):
My main issue is the lack of novelty and the issue that the results may be good due to “unfair decisions”.

Author response:
> My main issue is the lack of novelty and the issue that the results may be good due to “unfair decisions”. Mainly, the authors need to clarify if both the proposed model and the video-llama use the same encoders or not. If they don’t use the same encoders, the experiments should be rerun to have a consistent setting with the same encoders.
  - As explained for weaknesses 3 to 6, we believe our experiments are fair and the required comparison point (including different encoders and LoRA) can be found in Tables 8 and 9.

Reviewer comment (Experiments):
Mainly, the authors need to clarify if both the proposed model and the video-llama use the same encoders or not. If they don’t use the same encoders, the experiments should be rerun to have a consistent setting with the same encoders.

Author response:
> Mainly, the authors need to clarify if both the proposed model and the video-llama use the same encoders or not. If they don’t use the same encoders, the experiments should be rerun to have a consistent setting with the same encoders.
  - As explained for weaknesses 3 to 6, we believe our experiments are fair and the required comparison point (including different encoders and LoRA) can be found in Tables 8 and 9.

Reviewer comment (Experiments):
Moreover, the authors need to provide evidence that even when not fine-tuning the LLM and using consistent encoders with video-llama, their causal Q-former and synchronization module helps. Right now, the boost in performance could be attributed entirely to fine tuning the LLM.

Author response:
> Moreover, the authors need to provide evidence that even when not fine-tuning the LLM and using consistent encoders with video-llama, their causal Q-former and synchronization module helps. Right now, the boost in performance could be attributed entirely to fine tuning the LLM.
  - As explained for weaknesses 5 and 6, we’d like to further emphasise that 1). Incorporating speech and using the speech encoder is one of the key contributions. 2). The improvements using synchronisation and causal Q-Former is shown in Table 8 and 9. 3). It is also worth pointing out that Video-LLaMA used the entire 2.5M WebVid videos to train the video Q-Former which is 10 times more video data than what we used for training.

Reviewer comment (Experiments):
The authors should run the Video-LLAMA also on the task marked “-”, as mentioned in my previous comments. However, as I may be wrong here, I ask for clarifications for why this can not be done, if it can really not be done.

Author response:
> The authors should run the Video-LLAMA also on the task marked “-”, as mentioned in my previou comments. However, as I may be wrong here, I ask for clarifications for why this can not be done, if it can really not be done.
  - Please find our explanation for weakness 7 on this problem.

Reviewer comment (Experiments):
If these issues are addressed (mainly the fairness comparison), I think this paper is a good one, and even if it does not have a lot of novelty, it proposes a fairly simple extension that can obtain a significant boost in performance. But as it stands right now, there seems to be an unfair comparison and also lack of novelty and the paper is not good enough for being accepted.

Author response:
> If these issues are addressed (mainly the fairness comparison), I think this paper is a good one, and even if it does not have a lot of novelty, it proposes a fairly simple extension that can obtain a significant boost in performance. But as it stands right now, there seems to be an unfair comparison and also lack of novelty and the paper is not good enough for being accepted.
  - Thank you for the positive comments, and hope our explanation resolves your concerns.

Reviewer comment (Novelty):
The novelty of FAVOR is limited. Overall, it bears similarities to Video-LLaMA. The method of integrating speech is not uniquely designed, and the use of sliding windows is not particularly novel.

Author response:
- We would like to first reiterate the novelty of the proposed method. To the best of our knowledge, FAVOR is the first model that integrates speech audio events and video inputs, which is the key novelty of this paper. Although speech is an essential and common element in videos, it is often ignored in video studies outside the speech community. 

  - It is also challenging for LLM to incorporate speech input alongside video and this is the most salient aspect that sets FAVOR apart from other existing work. For example, the AVSR task used in our paper requires fine-grained and synchronised modelling of both audio and lip movements to achieve robust speech recognition in highly noisy acoustic environments. This motivates some unique designs of the FAVOR structure, including fine-grained audio-visual synchronisation, the causal Q-Former structure, and the diversity loss etc. 

  - Another important aspect of FAVOR that distinguishes it from other methods is the support of randomly paired audio and video data for training. This methodological innovation not only enhances the model's versatility but also fosters a more balanced integration of the two modalities. This enables FAVOR to perform audio-visual co-reasoning tasks in this paper, including ISQA and AVM. Moreover, to the best of our knowledge, FAVOR is the first model that is able to perform the open-ended audio-visual-speech-text questions demonstrated in Fig.9 to 12 in the appendix.

  - In particular, we’d like to clarify the following superior aspects of FAVOR to Video-LLaMA. 
    - FAVOR is able to understand speech input which is unable to be handled by Video-LLaMA. 
    - Incorporating speech input resulted in a significantly different structure design of FAVOR compared to Video-LLaMA, including synchronisation, sliding window operation and causal Q-Former.
    - In contrast to sampling a fixed number of frames regardless of the video length in Video-LLaMA, FAVOR performs fine-grained modelling and the sliding window allows more output tokens for longer videos, which leads to significantly better performance.
__As a result, Video-LLaMA could not perform any of the examples shown in Fig. 6 to 12.__

Reviewer comment (Novelty):
The audio-visual benchmark makes a limited contribution since the majority of the tasks and datasets are adopted from other sources, and there is only one evaluation dataset for each task.

Author response:
- The main aim of single-modal tasks in AVEB is to show the competitive ability of FAVOR in each modality, including speech, audio, image and video, and for this purpose, one representative task for each modality was selected.

Reviewer comment (Experiments):
As for Tables 2 and 3, more models can be added in comparison, and it is not clear whether each model is evaluated in a zero-shot setting in each task, as the training datasets are only mentioned in the text. Furthermore, the SOTA performance in a fair comparison setting for each task is not specified in the tables, making it challenging to assess the effectiveness of FAVOR.

Author response:
- The categories of the tasks are listed in the following table.
  - 
| Type | Tasks |
| --- | --- |
| Zero-shot | IC, VQA, OCR, ISQA, AVM (audio-video matching), AVSSD |
| Trained | ASR, AC, Video QA, AVSR, AVSD |

  - Image tasks (caption, OCR and VQA) were set to be zero-shot for a fair comparison with the original InstructBLIP, and AVSSD and audio-video matching as zero-shot tasks for a fair comparison with Video-LLaMA. We further finetuned the InstructBLIP on the same training set as FAVOR to eliminate the influence of video data choices.

Reviewer comment (Novelty):
The diversity loss does not introduce a novel concept as it essentially constitutes a similarity loss applied across features. In addition, it is unclear why, in the second point of the main contributions, the diversity training loss can enable the causal Q-Former to efficiently handle audio-visual sequences from a small number of training examples, as this is not evident from the experimental results. From Table 9, the diversity loss does not necessarily lead to improvements on every task; in some cases, there are even performance degradations. To prove that diversity loss enhances model learning, applying it to other Q-Former-based models should also yield improvements.

Author response:
- We thank the reviewer for pointing out the confusion of the claim of diversity loss and have removed the claim in the revised paper. The diversity loss is mainly used to encourage diverse information to be extracted, which is useful for long videos as reflected in video QA and AVSD tasks.

Reviewer comment (Experiments):
The recently introduced VALOR-32K dataset, which centers around audio-visual captioning, could be included in AVEB. Integrating this dataset and associated task could substantially enhance the comprehensiveness of the benchmark and facilitate a more thorough evaluation of audio-visual models.

Author response:
- We appreciate the reviewer’s suggestion of adding VALOR-32k as a task, and we’d like to add the following results for both zero-shot and fine-tuned performance of FAVOR on VALOR-32k using METEOR (M), Rouge-L (R) and CIDEr (C). We were only able to download __10%__ of the training set from YouTube given the limited time, and we finetuned FAVOR and Video-LLaMA on that __10% of data only__. We also added VALOR as a baseline evaluated on AVEB for tasks covered by VALOR, and will also include the suggested references in the revised version. 
  - 
| System | Zero-shot (M/R/C) (↑) | Fine-tuned (M/R/C) (↑) |
| --- | --- | --- |
| VALOR-32k base | - | 14.8 / 30.8 / 55.7 |
| Video-LLaMA (10% data) | 10.7 / 15.3 / 1.2 | 10.9 / 23.0 / 21.3 |
| FAVOR (10% data) | 8.8 / 19.2 / 15.3 | 14.2 / 29.4 / 46.9 |

  - In the zero-shot case, Video-LLaMA tends to generate long paragraphs of text even under the instruction of generating short sentence responses. This resulted in extremely low CIDEr scores compared to FAVOR which closely follows the instruction and generate concise responses. __This is another advantage of FAVOR over Video-LLaMA.__

Reviewer comment (Experiments):
More multi-modal models should be included in the experiments, such as VALOR and VAST.

Author response:
- We appreciate the reviewer’s suggestion of adding VALOR-32k as a task, and we’d like to add the following results for both zero-shot and fine-tuned performance of FAVOR on VALOR-32k using METEOR (M), Rouge-L (R) and CIDEr (C). We were only able to download __10%__ of the training set from YouTube given the limited time, and we finetuned FAVOR and Video-LLaMA on that __10% of data only__. We also added VALOR as a baseline evaluated on AVEB for tasks covered by VALOR, and will also include the suggested references in the revised version. 
  - 
| System | Zero-shot (M/R/C) (↑) | Fine-tuned (M/R/C) (↑) |
| --- | --- | --- |
| VALOR-32k base | - | 14.8 / 30.8 / 55.7 |
| Video-LLaMA (10% data) | 10.7 / 15.3 / 1.2 | 10.9 / 23.0 / 21.3 |
| FAVOR (10% data) | 8.8 / 19.2 / 15.3 | 14.2 / 29.4 / 46.9 |

  - In the zero-shot case, Video-LLaMA tends to generate long paragraphs of text even under the instruction of generating short sentence responses. This resulted in extremely low CIDEr scores compared to FAVOR which closely follows the instruction and generate concise responses. __This is another advantage of FAVOR over Video-LLaMA.__

Reviewer comment (Evaluation):
The importance and impact of the storytelling fine-tuning set are not explicitly explained. If fine-tuning on this dataset does not affect the benchmark performance, the reason for fine-tuning is unclear.

Author response:
- Storytelling data was used as a further fine-tuning stage to foster an extensive mixing of audio-visual narratives. This primarily aims to guarantee a better demonstration quality for open-ended questions, e.g. Fig. 9 to Fig. 12, that can only evaluated qualitatively. This stage is crucial to prevent the model’s tendency to perform a specific training set task when answering the open-ended question.

Reviewer comment (Experiments):
Examples illustrating the differences in the model before and after storytelling fine-tuning are not provided.

Author response:
- Storytelling data was used as a further fine-tuning stage to foster an extensive mixing of audio-visual narratives. This primarily aims to guarantee a better demonstration quality for open-ended questions, e.g. Fig. 9 to Fig. 12, that can only evaluated qualitatively. This stage is crucial to prevent the model’s tendency to perform a specific training set task when answering the open-ended question.

Reviewer comment (Reproducibility):
The number of parameters that need to be trained in FAVOR is not stated. The computational resources (GPUs) used for training the model and the training time with these resources are also not disclosed.

Author response:
- We have 30M parameters to be trained in FAVOR. Our best-performing model used 16 A100 GPUs to train for 48 hours.

Reviewer comment (Experiments):
It is unclear how InstructBLIP is evaluated on the cross-modal tasks AVSD and AVSSD, given that it cannot handle audio input. It is not specified whether it had any access to audio input information during evaluation.

Author response:
- InstructBLIP is evaluated without audio for those two tasks. However, by examining the data, the majority of audio events can be inferred from the visual modality, and that is why by fine-tuning on our data, InstructBLIP had some performance gain. This also demonstrated the necessity of modelling speech input, as non-speech audio alone in many cases can be inferred directly from the video.

Reviewer comment (Novelty):
The overall contribution and technical novelty do not meet the bar of ICLR. Audio-visual large model is not a new idea. Using feature syncing and concatenation of different modalities, causal self-attention, and sliding windows are well-known technologies.

Author response:
- It is important to emphasise that FAVOR is the first multimodal LLM that integrates speech and non-speech audio with video inputs, which is the key novelty of this paper. Although it might not be obvious to researchers outside of the speech and audio community, incorporating speech input along with other modalities is very challenging and important for LLM, which distinguishes FAVOR from other existing work. The fact that few studies support speech input in videos in an end-to-end fashion further corroborates this challenge. 

  - The necessity of modelling fine-grained temporal information in many speech-centric audio-visual tasks can be exemplified in the AVSR task in our paper where both audio and lip movements are needed to achieve robust speech recognition in highly noisy acoustic environments. This motivates some unique designs of the FAVOR structure, including fine-grained audio-visual synchronisation, the causal Q-Former structure, and the diversity loss etc. 

  - Meanwhile, in contrast to other existing work that uses video and audio from the same source to train, FAVOR adopts a novel strategy of employing randomly paired audio and video streams in its training regime. This increases versatility and achieves a better balance between the two modalities. It also enables FAVOR to perform audio-visual co-reasoning, including ISQA and AVM. Moreover, to the best of our knowledge, FAVOR is the first single model that is able to perform the open-ended audio-visual-speech-text questions demonstrated in Fig. 9 to 12 in the appendix.

Reviewer comment (Experiments):
The original self-attention unit also has the ability to attend to the contextual information (including previous frames). What is the additional gain to have another causal self-attention unit in the network? The motivation of such a design is unconvincing from the paper. Where does the claim "what happens next questions are sometimes difficult to learn using only the positional embeddings" come from? Are there solid studies, experiments, or data points supporting this claim? Please clarify.

Author response:
- The improvement of using causal attention is shown in Table 4. The Video QA task uses the NEXT-QA containing mainly temporal correlation questions, i.e. what happens next. Using causal attention improved the performance by a 6.5% absolute accuracy compared to merely using positional encoding. A vast amount of video training labels only concern one or two frames and do not emphasise the causal relation, hence structural enforcement is particularly useful to facilitate the model to pay attention to the causal relation.

  > (2) For the diversity loss, isn't the current equation also pushing the same feature far away (when i = j)? I don't feel the design is correct if there are no additional constraints on it. Isn't such design encouraging random features?
  - In our implementation, we excluded the diagonal terms during training and hence i=j case is actually not included in the total loss. Thank you for noticing this typo and we have made it clear in Eqn. 4.

Reviewer comment (Theory):
For the diversity loss, isn't the current equation also pushing the same feature far away (when i = j)? The design does not seem correct if there are no additional constraints on it. Isn't such a design encouraging random features?

Author response:
- The diversity loss is designed mainly to avoid the model focusing on keyframes with similar and repeated information in long videos and to extract more diverged features. Such benefits are most obviously reflected in video understanding tasks including Video QA (Acc. increased from 47.1% to 49.3%) and AVSD (Acc. increased from 53.9% to 54.5%) compared to the model without diversity loss. It is reasonable the diversity loss is not useful for most image or audio tasks since they don’t need a loss to encourage feature diversity, and hence, we received mixed results on image and audio tasks. The key novelty of this paper is to enable speech understanding in video-based LLMs and the diversity loss is just an associative design. We have weakened our claim on the diversity loss in the revised paper.

Reviewer comment (Writing):
The paper uses "fine-grained" in the title and abstract, but there isn't anything really related to "fine-grained" in the main paper. What does "fine-grained" here stand for?

Author response:
- The definition of “fine-grained” is the fine-grained temporal resolution, which is particularly crucial for tasks with speech and video inputs such as AVSR as shown in Table 4. I have revised the paper to make this clear.

Reviewer comment (Evaluation):
From the results, FAVOR yields over 20% improvement, but where the improvement comes from is not discussed in the paper. A simple claim that "the fine-grained causal modelling of video" is not sufficient to answer the question; more details should be discussed.

Author response:
- The improvements come from two major aspects: fine-grained modelling and causal attention. The influence of temporal resolution is illustrated in the right-most plot of Fig. 3. The influence of causal attention (as in the answer to point 2) is illustrated in Table 4. We will provide a more detailed explanation in section 5.3 in the revised paper.

Reviewer comment (Novelty):
The statement "... and to capture the correlation between what it 'hears' and 'sees'. Such tasks were almost infeasible for any other audio-visual models so far, since they were unable to understand both speech and non-speech sounds and did not model the audio-visual correlations in fine-grain" needs supporting evidence. First, solid datapoints on the failure of other audio-visual models should be provided (add citations). Second, why FAVOR can handle this task should be better discussed.

Author response:
- The following table summarises the existing multi-modal LLM handling video inputs to our best knowledge
  | Model | Speech | Audio | Video | Paper Links |
  | --- | --- | --- | --- | --- |
  | Video-LLaMA | No | Yes | Yes | https://arxiv.org/abs/2306.02858 |
  | PandaGPT | No | Yes | Yes | https://arxiv.org/abs/2305.16355 |
  | VideoChatGPT | No | No | Yes | https://arxiv.org/abs/2306.05424 |
  | Macaw-LLM | No | Yes | Yes | https://arxiv.org/abs/2306.09093 |
  | InstructBLIP | No | No | Yes | https://arxiv.org/abs/2305.06500 |
  | AVLFormer | No | Yes | Yes | https://arxiv.org/abs/2303.15616 |
  | VALOR | No | Yes | Yes | https://arxiv.org/abs/2304.08345 |
  | VAST | External subtitles | Yes | Yes | https://arxiv.org/abs/2305.18500 |
  - __None of them is able to perform the tasks shown in Fig. 6 to 12.__ FAVOR is able to understand both general audio (including speech and non-speech) and video and is trained to handle mismatched audio and video. Hence FAVOR is able to perform those co-reasoning tasks.

Reviewer comment (Reproducibility):
The paper provides certain examples of FAVOR including acoustic audios or music rather than speeches only, but it is unclear how FAVOR models them. Do we need to model them separately from speech, or do we just send them to the model as is together with speeches? What about the tones, rhythms, etc.?

Author response:
- FAVOR captures both speech and audio (including music) using a single Whisper encoder, which takes a single audio stream as input. Although Whisper has the ability to distinguish both speech and non-speech audio [1], it is not a music-specialised model and may not be able to achieve high performance on tasks of tones and rhythms. This is indeed an interesting direction to explore in the future.

Reviewer comment (Presentation):
Fig 2 is unclear. Is the input query representing the "q1,...,qn" in Eq.(2), and the feature sequence representing the "h_t^AV, ... h_(t+k)^AV" in Eq.(2)?

Author response:
- Thanks for pointing this out. The red feature corresponds to $\mathbf{h}^\text{V}$ and the yellow feature corresponds to $\mathbf{h}^\text{A}$. The concatenated red and yellow features correspond to $\mathbf{h}^\text{AV}$. Input query q is not shown in the figure, and the output query (blue strips) represents $\mathbf{h}^\text{Q}$

Reviewer comment (Experiments):
What is the storytelling fine-tuning set designed for? Why do we need such a set? Is it for training or benchmarking? If for benchmarking, what is it benchmarking for?

Author response:
- The storytelling is used only as a further fine-tuning stage to encourage a thorough mixture of audio-visual descriptions. This is mainly to guarantee a better demonstration quality for open-ended questions, e.g. Fig. 9 to Fig. 12, that are unable to be quantified. This stage is found crucial to prevent the model’s tendency to perform a specific training set task when answering the open-ended question.

Reviewer comment (Writing):
The present study's primary contribution pertains to the audio-visual joint representation, which lacks a clear definition of the term "fine-grained." It is uncertain how this term is used in the context of this paper, and further clarification is required to better understand the audio-visual joint representation in question.

Author response:
>The present study's primary contribution pertains to the audio-visual joint representation, which lacks a clear definition of the term "fine-grained." It is uncertain how this term is used in the context of this paper, and further clarification is required to better understand the audio-visual joint representation in question.
> According to the network design, the term "fine-grained" may denote fine-grained temporal resolution, where the temporal synchronization module is employed to synchronize visual and audio signals with a resolution of 0.5 seconds. However, the ablation of the temporal resolution as well as the rationale behind this synchronization have not been clearly explicated. It is essential to understand the underlying reasons for the setting of the temporal resolution and the motivation behind the synchronization, as it serves as a crucial aspect of the network design. Hence, further elaboration is required to provide a comprehensive understanding of the fine-grained temporal resolution and its relevance to the proposed network design.
  - “Fine-grained” here refers to a fine temporal resolution. We will make this clear in the paper. 
  - Fine-grained modelling and synchronisation are particularly designed for the joint representation learning of speech and video, as speech contains fine-grained and temporally synchronised information with the video, motivating us to adopt this model design. This is reflected by the 10% relative improvements achieved on the AVSR task in the ablation studies in Table 4. 
  - Moreover, different resolutions are also studied in Fig. 3 for both sliding window size and frames-per-second (i.e. temporal resolution).

Reviewer comment (Experiments):
According to the network design, the term "fine-grained" may denote fine-grained temporal resolution, where the temporal synchronization module is employed to synchronize visual and audio signals with a resolution of 0.5 seconds. However, the ablation of the temporal resolution as well as the rationale behind this synchronization have not been clearly explicated. It is essential to understand the underlying reasons for the setting of the temporal resolution and the motivation behind the synchronization, as it serves as a crucial aspect of the network design. Hence, further elaboration is required to provide a comprehensive understanding of the fine-grained temporal resolution and its relevance to the proposed network design.

Author response:
>The present study's primary contribution pertains to the audio-visual joint representation, which lacks a clear definition of the term "fine-grained." It is uncertain how this term is used in the context of this paper, and further clarification is required to better understand the audio-visual joint representation in question.
> According to the network design, the term "fine-grained" may denote fine-grained temporal resolution, where the temporal synchronization module is employed to synchronize visual and audio signals with a resolution of 0.5 seconds. However, the ablation of the temporal resolution as well as the rationale behind this synchronization have not been clearly explicated. It is essential to understand the underlying reasons for the setting of the temporal resolution and the motivation behind the synchronization, as it serves as a crucial aspect of the network design. Hence, further elaboration is required to provide a comprehensive understanding of the fine-grained temporal resolution and its relevance to the proposed network design.
  - “Fine-grained” here refers to a fine temporal resolution. We will make this clear in the paper. 
  - Fine-grained modelling and synchronisation are particularly designed for the joint representation learning of speech and video, as speech contains fine-grained and temporally synchronised information with the video, motivating us to adopt this model design. This is reflected by the 10% relative improvements achieved on the AVSR task in the ablation studies in Table 4. 
  - Moreover, different resolutions are also studied in Fig. 3 for both sliding window size and frames-per-second (i.e. temporal resolution).

Reviewer comment (Novelty):
I am seeking clarification about the contribution of the proposed framework design. What sets it apart from previous methods? The act of incorporating additional modalities into a multimodal LLM appears to be a trivial implementation. Furthermore, the Q-former has previously been proposed for multimodal fusion. Thus, I am curious to know what novel aspects this framework design offers.

Author response:
> I am seeking clarification about the contribution of the proposed framework design. What sets it apart from previous methods? The act of incorporating additional modalities into a multimodal LLM appears to be a trivial implementation. Furthermore, the Q-former has previously been proposed for multimodal fusion. Thus, I am curious to know what novel aspects this framework design offers.
  - As stated above, the incorporation of speech motivates the design of a frame-level (i.e. fine-grained) audio-visual feature fusion before sending it to the Q-Former. This structural consideration has not been explored by any other audio-visual LLM in literature to our best knowledge. Hereby we provide the following table summarising some existing methods (up to the ICLR 2024 submission deadline) that deal with video and make the contrast to FAVOR.

| Model | Speech | Audio | Video | Paper Links |
| --- | --- | --- | --- | --- |
| Video-LLaMA | No | Yes | Yes | https://arxiv.org/abs/2306.02858 |
| PandaGPT | No | Yes | Yes | https://arxiv.org/abs/2305.16355 |
| VideoChatGPT | No | No | Yes | https://arxiv.org/abs/2306.05424 |
| Macaw-LLM | No | Yes | Yes | https://arxiv.org/abs/2306.09093 |
| InstructBLIP | No | No | Yes | https://arxiv.org/abs/2305.06500 |
| AVLFormer | No | Yes | Yes | https://arxiv.org/abs/2303.15616 |
| VALOR | No | Yes | Yes | https://arxiv.org/abs/2304.08345 |
| VAST | External subtitles | Yes | Yes | https://arxiv.org/abs/2305.18500 |
__Note that none of the above systems is able to perform example tasks shown in the appendix (i.e. Fig. 6 to 12).__

Reviewer comment (Novelty):
The proposed audio-visual evaluation benchmark appears to be a composite of existing multimodal benchmarks. It is suggested that the term "propose" be replaced with a more appropriate term. Moreover, the benchmark in question, as a fine-grained multimodal benchmark, lacks some existing fine-grained audio-visual benchmarks, such as FAVDBench: Fine-grained Audible Video Description (CVPR 23).

Author response:
> The proposed audio-visual evaluation benchmark appears to be a composite of existing multimodal benchmarks. It is suggested that the term "propose" be replaced with a more appropriate term. Moreover, the benchmark in question, as a fine-grained multimodal benchmark, lacks some existing fine-grained audio-visual benchmarks, such as FAVDBench: Fine-grained Audible Video Description (CVPR 23).
  - We will remove the word “propose”. However, we would like to clarify that the audio-visual co-reasoning tasks, including image-spoken QA (ISQA) and audio-visual matching (AVM), are indeed newly proposed tasks. We also thank the reviewers for pointing out the additional FAVDBench data, and we have included both zero-shot learning and fine-tuned results in the modified paper as follows using BLEU 1&4 (B1 B4) and METEOR (M): 
  - 
| System | Zero-shot (B1/B4/M) (↑) | Fine-tuned (B1/B4/M) (↑) |
| --- | --- | --- |
| Video-LLaMA | 20.8 / 2.4 / 15.0 | 39.4 / 6.5 / 16.5 |
| FAVOR | __28.2 / 3.0 / 15.2__ | __44.2 / 10.9 / 19.1__ |
  - __We’d like to additionally emphasise that FAVOR is also able to capture speech content in those videos and attribute them to the speaker, which is even missing in the reference caption (e.g. video XNw23PiqlU0). Those useful outputs actually had a negative effect on the metrics above. It also showed how speech is understudied and how necessary to add speech in audio-visual LLMs.__

Reviewer comment (Experiments):
The author of the paper expounds on the utilization of speech and speech-video interactions. In speech-videos, two options exist for inputting speech audios to the network. These options include direct inputting of speech audios into the network or converting speech to text before the multi-modal fusion process. However, the author does not provide a comparative analysis of the two strategies.

Author response:
> The author of the paper expounds on the utilization of speech and speech-video interactions. In speech-videos, two options exist for inputting speech audios to the network. These options include direct inputting of speech audios into the network or converting speech to text before the multi-modal fusion process. However, the author does not provide a comparative analysis of the two strategies. Furthermore, the ASR in audio-visual inputs is worse than audio-only, as demonstrated in Table 2. This result suggests that using the latter option may be more favorable. Lastly, the size of Whisper Large-v2 is not available in the paper.
  - We’d like to emphasise the following advantages of FAVOR which models general audio (including speech) and vision using a single unified model. 

    - Having an external automatic speech recognition (ASR) system to convert speech to text first is significantly more complicated, compared to FAVOR which is a single end-to-end model. 

    - All realistic ASR systems must contain an LM component, even an LLM component (e.g. [1, 2]), to obtain accurate speech recognition results. Having an external ASR to first transcribe the video into subtitles becomes a chicken-egg problem when the goal is to extend the LLM to handle multiple modalities. 

    - It’s not trivial for recent ASR approaches to provide the time information of each word, which makes it difficult and tedious to explicitly align the words with the visual frames and therefore not possible to handle cases where speech-video synchronisation matters. 

    - In addition to texts that can be transcribed using an external ASR system, there are other types of auditory information in speech that are also critical in video understanding, such as speaker characteristics, emotion, gender etc. Each of these usually requires a separate system to extract. The outputs of the separate systems then need to be aligned with the ASR outputs and the visual frames correctly, which is not trivial. However, in the framework of FAVOR, all these can be unified in a fully end-to-end fashion using a single system. 

- The size of Whisper large-v2 is 1550M and we will add this to our revised paper.

[1] Yassir Fathullah et al. “Prompting Large Language Models with Speech Recognition Abilities”, arXiv:2307.11795, 2023.

[2] Paul K. Rubenstein et al.“AudioPaLM: A Large Language Model That Can Speak and Listen”, arXiv:2306.12925, 2023

Reviewer comment (Evaluation):
The ASR in audio-visual inputs is worse than audio-only, as demonstrated in Table 2. This result suggests that using the latter option (converting speech to text before the multi-modal fusion process) may be more favorable.

Author response:
> The author of the paper expounds on the utilization of speech and speech-video interactions. In speech-videos, two options exist for inputting speech audios to the network. These options include direct inputting of speech audios into the network or converting speech to text before the multi-modal fusion process. However, the author does not provide a comparative analysis of the two strategies. Furthermore, the ASR in audio-visual inputs is worse than audio-only, as demonstrated in Table 2. This result suggests that using the latter option may be more favorable. Lastly, the size of Whisper Large-v2 is not available in the paper.
  - We’d like to emphasise the following advantages of FAVOR which models general audio (including speech) and vision using a single unified model. 

    - Having an external automatic speech recognition (ASR) system to convert speech to text first is significantly more complicated, compared to FAVOR which is a single end-to-end model. 

    - All realistic ASR systems must contain an LM component, even an LLM component (e.g. [1, 2]), to obtain accurate speech recognition results. Having an external ASR to first transcribe the video into subtitles becomes a chicken-egg problem when the goal is to extend the LLM to handle multiple modalities. 

    - It’s not trivial for recent ASR approaches to provide the time information of each word, which makes it difficult and tedious to explicitly align the words with the visual frames and therefore not possible to handle cases where speech-video synchronisation matters. 

    - In addition to texts that can be transcribed using an external ASR system, there are other types of auditory information in speech that are also critical in video understanding, such as speaker characteristics, emotion, gender etc. Each of these usually requires a separate system to extract. The outputs of the separate systems then need to be aligned with the ASR outputs and the visual frames correctly, which is not trivial. However, in the framework of FAVOR, all these can be unified in a fully end-to-end fashion using a single system. 

- The size of Whisper large-v2 is 1550M and we will add this to our revised paper.

[1] Yassir Fathullah et al. “Prompting Large Language Models with Speech Recognition Abilities”, arXiv:2307.11795, 2023.

[2] Paul K. Rubenstein et al.“AudioPaLM: A Large Language Model That Can Speak and Listen”, arXiv:2306.12925, 2023

Reviewer comment (Reproducibility):
The size of Whisper Large-v2 is not available in the paper.

Author response:
> The author of the paper expounds on the utilization of speech and speech-video interactions. In speech-videos, two options exist for inputting speech audios to the network. These options include direct inputting of speech audios into the network or converting speech to text before the multi-modal fusion process. However, the author does not provide a comparative analysis of the two strategies. Furthermore, the ASR in audio-visual inputs is worse than audio-only, as demonstrated in Table 2. This result suggests that using the latter option may be more favorable. Lastly, the size of Whisper Large-v2 is not available in the paper.
  - We’d like to emphasise the following advantages of FAVOR which models general audio (including speech) and vision using a single unified model. 

    - Having an external automatic speech recognition (ASR) system to convert speech to text first is significantly more complicated, compared to FAVOR which is a single end-to-end model. 

    - All realistic ASR systems must contain an LM component, even an LLM component (e.g. [1, 2]), to obtain accurate speech recognition results. Having an external ASR to first transcribe the video into subtitles becomes a chicken-egg problem when the goal is to extend the LLM to handle multiple modalities. 

    - It’s not trivial for recent ASR approaches to provide the time information of each word, which makes it difficult and tedious to explicitly align the words with the visual frames and therefore not possible to handle cases where speech-video synchronisation matters. 

    - In addition to texts that can be transcribed using an external ASR system, there are other types of auditory information in speech that are also critical in video understanding, such as speaker characteristics, emotion, gender etc. Each of these usually requires a separate system to extract. The outputs of the separate systems then need to be aligned with the ASR outputs and the visual frames correctly, which is not trivial. However, in the framework of FAVOR, all these can be unified in a fully end-to-end fashion using a single system. 

- The size of Whisper large-v2 is 1550M and we will add this to our revised paper.

[1] Yassir Fathullah et al. “Prompting Large Language Models with Speech Recognition Abilities”, arXiv:2307.11795, 2023.

[2] Paul K. Rubenstein et al.“AudioPaLM: A Large Language Model That Can Speak and Listen”, arXiv:2306.12925, 2023